Calibrating Software Cost Models to Department of Defense Databases - a Review of Ten Studies

There are many sophisticated parametric models for estimating the size, cost, and schedule of software projects. In general, the predictive accuracy of these models is no better than within 25 percent of actual cost or schedule, about one half of the time (Thibodeau, 1981; IIT Research Institute, 1988). Several authors assert that a model's predictive accuracy can be improved by calibrating (adjusting) its default parameters to a specific environment (Kemerer, 1987; Van Genuchten and Koolen, 1991; Andolfi et al. 1996). This paper reports the results of a long-term project that tests this assertion. From 1995 to 1997, masters students at the Air Force Institute of Technology calibrated selected software cost models to databases provided by two Air Force product centers. Nine parametric software models (REVIC, SASET, PRICE-S, SEER-SEM, SLIM, SOFTCOST, CHECKPOINT, COCOMO II, and SAGE) were calibrated. Data from the product centers were extracted and stratified for specific software estimation models, calibrated to specific environments, and validated using hold-out samples. The project was limited to software development cost or effort, although the same procedures could be used for size, schedule, or other estimating applications. Results show that calibration does not always improve a model's predictive accuracy. Although one model which uses function points did show significantly improved accuracy (Mertes, 1996), the results could not be replicated on another database (Marzo, 1997).